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SANTRAUKA 

Europos universitetij vaidmuo atliekant ins- 
titucin j politiniq irsocialiniq sistemq vertini- 
mq per pastaruosius 30 metq zymiai isaugo. 
Daugiausia jie buvo atsakingi uz jvertina- 
mojo tyrimo metodij ir tam tikrq apsaugos 
garantij pateikimq. Siame straipsnyje patei- 
kiame pavyzdj, kaip aukstojo mokslo siste- 
moje taikomos jvertinimo procedures buvo 
panaudotos vertinant mokymo ir uzimtumo 
programas, skirtas jgyvendinti su lyciq lygy- 
be susijusios socialines politikos. Teikiamq 
paslaugij kokybei jvertinti pasitelkti kieky- 
biniai ir kokybiniai tyrimij metodai. Atsi- 
zvelgta tiek j personalo, tiek ir j programos 
dalyviij nuomon^. Straipsnyje aprasomi su 
kiekybinemis tyrimo priemonemis susije; 
procesai ir gauti rezultatai. Paaiskejo, kad 
programos dalyviij amzius ir akademinis 
laipsnis - pagrindiniai veiksniai, leidziantys 
susirasti darlxj ir suteikiantys pasitenkinim;j 
uzimtumo programa. 


1 Straipsnio autoriai dekoja Robert Stake uz pagalb^, pas- 
tabas ir pasiulymus. Taip pat norime padekoti Luz Villar uz 
pagalb^ rasant straipsnj anglp kalba. 


ABSTRACT 

Universities in Europe have been playing an 
increasingly important role in the institution¬ 
al evaluation of political and social systems 
for the last thirty years. Their major contri¬ 
bution to those processes of accountability 
has been to add methods and safeguards of 
evaluative research. In this paper we report 
an illustration of how evaluative procedures 
employed in higher education have been 
used for the evaluation of employment and 
training programs designed for the imple¬ 
mentation of social policies in gender equity. 
Both quantitative and qualitative research 
methods were used to evaluate the quality 
of the services provided both from staff and 
user perspectives. In this paper we report the 
process followed and the outcomes obtained 
with quantitative tools. Outcomes reveal that 
users’ age and grades are key factors both to 
get a job and to be pleased with the employ¬ 
ment program. 


1 The authors appreciate the assistance, comments and 
suggestions given by Robert Stake to this paper. We also 
thank Luz Villar for her assistance with the English writing. 
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PAGRINDINiy TERMINI^ APIBREZIMAI 

• Apmokymas - mokymosi procesas, skirtas 
pletoti bedarhiij jsidarbinimo mokejimus ku- 
rio pagrindinis tikslas - pagerinti ir pritaikyti 
gyvenime profesines zinias bei mokejimus. 

• Isorinis vertinimas - jvertinimo procesas, 
apimantis analizavimq, vertinimq ir pasiuly- 
mus, kuriuos pateikia su jvertinamu objektu 
tiesiogiai nesusijusi jstaiga. 

• Kiekybine metodologija - statistikos prie- 
moniij naudojimas tyrimuose. 

• Programos jvertinimas - informacijos, skir- 
tos programq kokybei, pasiekimams ir tru- 
kumams jvertinti, rinkimas, siekiant priimti 
sprendimus, susijusius su personalu ir pro¬ 
gram^ pasirinkusiais bendruomenes nariais. 
Taip norima pagerinti vertinamos programos 
rezultatus. 

• Socialine politika - valstybes naudojamos 
priemones, leidziancios kistis j pilietines vi- 
suomenes, ypac tq socialiniij grupiq, kurios 
susiduria su kulturiniais, socialiniais ar eko- 
nominiais sunkumais, reikalus. 

JVADAS 

Programos jvertinimq daugiausia sudaro 
procesai, strategijos ir veiksmai, skirti viesqjq 
paslaugij kokybei gerinti. Dauguma siq ver- 
tinamqjq procesq pirmiausia pradeti naudoti 
tam tikrose profesinese srityse ir tik veliau 
juos pereme ir savo veikloje placiai pritaike 
valdzios ir kitos jstaigos. 

Atskaitomybe (angl. accountability) - pa- 
grindine sritis, kur institucinis jvertinimas 
buvo pletojamas pastaruosius 30 metip At- 
skaitomybes strategy pletra atskleide, kad 
viestyq istaigij sprendimai ir veiksmai nuo 
tada buvo orientuoti j jij atsakomybeje esan- 
ciij procesij tobulinim^. Vienas pagrindiniij 
siekiij - teisinis ir intelektinis jgaliojimas 
skelbti apie padaliniij ir nariij, susijusiij su 


DEFINITIONS OF KEY TERMS 

• Training - process of learning set on employ- 
ability, outplacement and work update with the 
major purpose of increasing and adapting pro¬ 
fessional knowledge and skills throughout life. 

• External evaluation - evaluation process 
comprising the analysis, valuation and propo¬ 
sals rose by an agency with no direct relation 
to the evaluand. 

• Quantitative methodology - focused on 
the usage of statistical procedures to analyse 
reality. 

• Program evaluation - process to collect in¬ 
formation intended to value quality, achieve¬ 
ments and shortcomings of programs to sup¬ 
port decision makings about the personnel 
staff involved and the community attended in 
order to improve the outcomes of the program 
evaluated. 

• Social policies - a way the states employ to 
intervene in civil society, basically with social 
groups with cultural, social or economical 
disadvantages. 

INTRODUCTION 

By and large, program evaluation implies a set 
of processes, strategies and actions aimed at 
improving the quality of public services. Most 
of these evaluative processes were taken up 
initially in the practice of professional fields 
and, later, added and adapted extensively by 
governments and agencies. 

Accountability has been the major frame¬ 
work where institutional evaluation has been 
developed for the last thirty years. Since then, 
development of accountability strategies has 
demonstrated that decisions and actions taken 
by public institutions have been oriented to im¬ 
provement of the processes under their respon¬ 
sibility. One major purpose is the intellectual and 
legal mandate to publicise institutional activities 
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vertinama programa, institucin^ veiklq. Taip 
siekiama suprasti ir atsakyti j iskilusius klau- 
simus (Trow, 1996). 

Del tokio poziurio j atskaitomyb^ auks- 
tqjq mokyklq atliekami jvertinimai skiriasi 
savo forma pagal jstaigos, valdzios institu- 
cijiy ir mokymo interesus. Pavyzdziui, akre- 
ditavimo sistema yra viena vertinimo formq, 
kai pati jstaiga, orientuodamasi j kokybes 
standartus, isoriniams vertintojams pateikia 
informacijq apie destytojo veiklg.. Rezultatq 
vertinimas - gautq rezultatq vertinimas pa¬ 
gal veiklos rodiklius. Ekspertinis jvertinimas, 
kitaip dar vadinamas isoriniu, kai remiamasi 
ekspertij nuomone, issakyta atsakant j klau- 
simus apie vertinamq istaigq. Taip pat reike- 
tq pamineti ir jvertinamqjj tyrimq, kuriuo 
siekiama rezultatus aiskinti atsizvelgiant j 
jdetas pastangas ir aukstqjq mokyklij sqly- 
gas (El-Khawas, 1993; Simoneau, 1991; Cave 
irkt., 1997). 

fvertinamuoju tyrimu siekiama atlik- 
ti isorinj programq jvertinimq. | pagalbq 
pasitelkiami isoriniai ekspertai tiek is ver- 
tinamos programos srities, tiek ir jvertini- 
mo profesionalai. {vertinimas - sistemingas 
duomenq rinkimo ir analizavimo procesas. 
Po to siuloma, kaip pagerinti teikiamas 
paslaugas. 

Siame straipsnyje aptariamas isoriniij 
vertintojq atliktas moterims skirtos uzim- 
tumo-mokymo programos jvertinimas. Pa- 
grindinis jo tikslas - issiaiskinti, kaip pro¬ 
grama pagerina jsidarbinimq. jvertinimui 
pasitelkti universitetp tyrimuose naudojami 
duomenij rinkimo ir analizes metodai. Tai- 
gi atliktas kiekybinis tyrimas: personalui ir 
programos dalyviams buvo pateiktos anke- 
tos, kurhj rezultatai ir isvados leistp gerinti 
pad/} program^. 

Iniciatyv^ jvertinti program^ rode 
pati jstaiga, siekdama suzinoti, kokj po- 
veikj turi jp siuloma programa. Taip pat 
noreta issiaiskinti, kiek svarbu skirti vals- 
tybines lesas moterims darbo ir mokymo 


of the units and members of the program evalu¬ 
ated. They seek to explain, justify and clarify 
questions that have arisen (Trow, 1996). 

From this perspective of accountability, 
different variations have emerged in evalua¬ 
tions carried out by members of higher-educa¬ 
tion institutions, variants conceived according 
to institutional, governmental and instruc¬ 
tional interests. For example, the accredita¬ 
tion system, a way of assessment for which the 
institution provides information to external 
evaluators about instructors’ activities with 
reference to standards of quality; outcome as¬ 
sessment, that is, the assessment of outcomes 
obtained according to performance indicators; 
peer review assessment, or external evaluation 
based on experts opinions answering to ques¬ 
tions from the institution assessed; and evalu¬ 
ative research, with the intention of explaining 
outcomes in terms of inputs and conditions 
of higher-education institutions (El-Khawas, 
1993; Simoneau, 1991; Cave et al., 1997). 

It is from this last perspective, that of evalu¬ 
ative research, that this paper starts. Evaluative 
research here pursues the external evaluation 
of programs carried out by external experts 
both from the discipline of the program and 
from the evaluation held. The evaluation is 
carried out as a systematic process of data col¬ 
lection and analysis, followed by proposals for 
service improvement. 

In this paper, we discuss the evaluation of 
an employment-training program for women 
carried out by external evaluators. The major 
purpose of this evaluative enquiry is to know 
the impact of the program to improve employ- 
ability by using evaluation techniques for col¬ 
lecting data and subsequent analysis, which 
come out of university research. Thus, we have 
carried out a quantitative study by means of 
questionnaires, intended, on the one hand, 
to users of the program and, on the other, to 
staff personnel, in order to obtain findings and 
conclusions which contribute to the program 
improvement. 
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programq pletrai. Toks jvertinimas siame 
straipsnyje pateikiamas kaip pavyzdys, kad 
aukstajame moksle jau jsigalejusios tyrimq 
metodologijos ir susij^ procesai pletros bei 
tobulejimo tikslais taip pat gali buti nau- 
dojami atliekant isorinj jvertinimq kitose 
jstaigose. 


1 

NESALISKAS VIESOSIOS POLITIKOS 
JVERTINIMAS 

Musq jvertintoje „Uzimtumo mokykloje“ (pa- 
vadinimas pakeistas) siulomos jvairios mote- 
rims skirtos uzimtumo ir mokymo programos. 
Visos jos siejamos su tam tikromis Europos 
Sqjungos (ES) lygiu galimybiij programomis, 
jskaitant „Uzimtumo, socialiniij re i kali} ir lygiij 
galimybiij program^' (angl. Employment, So¬ 
cial Affairs and Equal Opportunities program, 
EQUAL), apimanciq „naujas idejas Europos 
uzimtumo strategijoje ir socialities jtraukties 
procese" 2 , bei keleQ kitij Europos socialinio 
fondo (ESF) lesomis finansuojamq progra¬ 
ms, pavyzdziui, „Naujos galimybes moterims“ 
(angl. New Opportunities for Women, NOW). 
Jomis „ieskoma budij, kaip paprasciau moteris 
pritraukti ar sugrqzinti j darbo rinkq“ 3 , ypac 
akcentuojant nesaliskumo politikcj. 

Jvertinimui pasirinkta koncepcija ir me- 
todologine struktura atitiko Bendrijos inicia- 
tyvos EQUAL lyciij politik^ 4 ir nepriestaravo 


2 Internetinis adresas: http://ec.europa.eu/employment_social/ 
equal/index_en.cfm. Puslapis aplankytas 2011 m. vasario 4 d. 

3 Internetinis adresas: http://ec.europa.eu/employment_social/ 
esf/discover/esf_en.htm. Puslapis aplankytas 2011 m. vasario 4 d. 

4 Internetinis adresas: http://europa.eu/legislation_ 

summaries/employment_and_social_policy/equality_ 
between_men_and_women/cl0237_en.htm. Puslapis 
aplankytas 2011 m. vasario 4 d. 


In the experience reported in this paper, the 
initiative of the evaluation emerged from the 
institution evaluated to find its programmatic 
impact. Also sought was the social and instruc¬ 
tional relevance of the public funds invested in 
the development of labour and training pro¬ 
grams for women. In this article this evalua¬ 
tion is presented as illustration of how research 
methodologies and processes well settled in 
higher education can be implemented on exter¬ 
nal evaluation and on other institutions with the 
aim of getting development and improvement. 

1 

EVALUATION OF PUBLIC POLICIES AS 
TO EQUITY 

The School for Employment (SE) (supposed 
name) that we evaluated has carried out a col¬ 
lection of employment and training programs 
in support of women. The roots of these pro¬ 
grams are in several European Union (EU) 
equity programs, such as the Employment, So¬ 
cial Affairs and Equal Opportunities program 
(EQUAL), bringing ‘ fresh ideas to the Europe¬ 
an Employment Strategy and the Social inclu¬ 
sion process’’ 2 and some others funded by the 
European Social Fund (ESF), like the New Op¬ 
portunities for Women (NOW). They “looked 
at ways in which women could more easily be 
brought into, or back into, the labour market,’’ 3 
with a clear emphasis on policies of equity. 

For this reason, the conceptual and meth¬ 
odological framework of this evaluation re¬ 
sponded to gender policies of the Commu¬ 
nity Initiative EQUAL Act 4 , the experience of 
the Spanish Administrative Unit for the ESF 


2 Available at: http://ec.europa.eu/employment_social/ 
equal/index_en.cfm. Site visited 4th February 2011. 

3 Available at: http://ec.europa.eu/employment_social/esf/ 
discover/esf_en.htm. Site visited 4th February 2011. 

4 Available at: http://europa.eu/legislation_summaries/ 
employment_and_social_policy/equality_between_men_ 
and_women/c 10237_en.htm. Site visited 4th February 
2011. 
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Ispanijos administracinio vieneto prie ESF 
(UAFSE 5 ) veiklai. UAFSE yra nacionalines 
valdzios institucija, atsakinga uz sip Europos 
fondij administravimq. Pasinaudota keliose ES 
valstybese narese sukurtomis gairemis ir prie- 
monemis, kurias ESF nurode Moterp ir vyrij 
lygybes gairese (angl. Roadmap for Equality 
between women and men) 6 . ES kontekstas cia 
labai svarbus del ES politikos jtakos, daromos 
Ispanijos viesajai politikai. Be to, atsizvelgta ir 
j kitq tarptautinip institucijq veiklq, is kuriij 
reiketp pamineti Tarptautin^ darbo organiza- 
cijq (angl. International Labour Organization, 
ILO), Lotynq Amerikos ir Karibq juros regiono 
ekonomikos komisijq (angl. the Economic Com¬ 
mission for Latin America and the Caribbean, 
ECLAC), Kanados tarptautines pletros agentu- 
rq (angl. Canadian International Development 
Agency, CIDA) ir Tarptautinj zemes ukio plet¬ 
ros fondq (angl. the International Fund for 
Agricultural Development, IFAD). Svarbi sip 
jstaigp veikla aktyvios darbo politikos sferoje, 
ypac su lycip lygybe susij^ klausimai. 

Didzioji dalis sip institucijp vykdomos 
veiklos atitinka tarptautinip jvertintojp re- 
komendacijas, pateiktas pastaraisiais metais. 
Nemazai mokslininkp, tiriancip viesojo admi- 
nistravimo tarnybp vaidmenj, jvertindami ly¬ 
gybes ir uzimtumo politikq (pvz., Barrer, 2005; 
Blake-Beard ir kt„ 2010; Dick, Hyde, 2006; 
Heery, 2006; Hite, 2007; Sweeney, 2004; Worts 
ir kt., 2007), nurodo sias tobulintinas sritis: 

• Atskira politikos kryptis turetp buti 
orientuota j pagalbq moterims, nurodyti jas 
remiancius fondus ir atrinkti lygias galimybes 
skatinancius projektus. Daugiau lesp reiketp 
skirti lygioms galimybems uztikrinti, ypac ly¬ 
cip integracijai. 

• Reiketp sekti ir jvertinti vykdomcj. vei- 
kkj, nes taip galima issiaiskinti, ar lesos buvo 


5 UAFSE yra sutrumpinimas ispanp kalba. Informacij^ ispanij 
kalba galima rasti apsilankius http://www.mtin.es/uafse/. 

6 Internetinis adresas: http://ec.europa.eu/employment_ 
social/esf/fields/woman_en.htm. Puslapis aplankytas 
2011 m. vasario 4 d. 


(UAFSE 5 ) as national authority in charge of the 
administration of these European funds. This 
framework used guides and tools developed 
by several EU states as found in the Roadmap 
for Equality between women and men for the 
ESF 6 . The EU context is highly relevant due to 
the high impact EU policies have on Spanish 
public policies. Other parts of our framework 
came from international institutions, in par¬ 
ticular the International Labour Organization 
(ILO), the Economic Commission for Latin 
America and the Caribbean (ECLAC) - both 
part of the UN, - the Canadian International 
Development Agency (CIDA), and the Inter¬ 
national Fund for Agricultural Development 
(IFAD). Our review has coped with issues 
these agencies have developed about active la¬ 
bour policies at large, with emphasis on a gen¬ 
der perspective. 

Many activities carried out by these agen¬ 
cies are in accordance with recommendations 
given by international evaluators for the last 
years. Moreover, a significant amount of schol¬ 
ars analysing the role played by public admin¬ 
istrations in the evaluation of equality and 
employment policies (e.g., Barrer, 2005; Blake- 
Beard et al„ 2010; Dick y Hyde, 2006; Heery, 
2006; Hite, 2007; Sweeney, 2004; Worts et al„ 
2007) keeps on remarking important needs of 
improvement, such as: 

• There should be policies addressed to 
women specifically, making visible funds for 
their support, in particular selecting projects 
promoting equality policies. Funds for equal¬ 
ity should be increased, in particular, for gen¬ 
der integration. 

• There should be follow-up and evalu¬ 
ation of actions in progress to find out effec¬ 
tiveness and sustainability of actions funded, 
durability over time, and effect over peoples 
fives, women in particular. 


5 UAFSE is its acronym in Spanish. Information in that 
language is available at http://www.mtin.es/uafse/. 

6 Available at: http://ec.europa.eu/employment_social/esf/ 
fields/woman_en.htm Site visited 4th February 2011. 
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panaudotos efektyviai, taip pat jvertinti veik- 
los tqstinumq ir jos poveikj zmoniij, ypac mo- 
terij, gyvenimui. 

• Viesosios institucijos, turincios patir- 
ties lygiq galimyhiij klausimais, turetij remti 
ir skatinti atlikti daugiau sistemingu tyrimu 
apie moterij padetj, t. y. vykdyti projektus, 
kuriais pirmiausia butq jvertinti moterij po- 
reikiai, moterys laikomos socialine grupe, 
turincia savituimj ir pazeidziamij vietij, ypac 
kai kalbama apie nedarbq. 

• Norint mazinti skirtumus tarp vyrij ir 
moterij, reiketij daugiau demesio skirti rodi- 
kliams, susijusiems su lyciij lygybe. 

• Projekte turetij dalyvauti ir buti tinka- 
mai apmokyti ne tik ekspertai, dirbantys lygiij 
galimybiij srityje, bet ir kiti zmones, galintys 
daryti itakcy Juos reiketij supazindinti su lygiij 
galimybiij aspekto integravimo j programas 
privalumais ir su atitinkamomis realizavimo 
priemonemis. 

• Didelis diferencijuotumas tarp moterij 
reikalauja kurti programas, kurios neleistij 
formuotis ribinems moterij grupems, turin- 
cioms nedideles jsidarbinimo galimybes. As- 
meniniai pastebejimai ar uzrasai turetij buti 
kertiniai akmenys, padedantys moterims sek- 
mingai jsidarbinti. 

• Reikia skatinti moterij verslurmj, t. y. 
skirti lesij jmonems kurti, vadovavimui gerin- 
ti, jkvepti pradedancias verslininkes pasitiketi 
savo jegomis, jtraukti specialistus, remiancius 
moterims priklausancias jmones. 

• Reikia iniciatyvij, padedanciij suderin- 
ti darb;j ir seim;j; reiketij skatinti verslininkus 
jdarbinti zmones vienodomis s:jlygomis, ne- 
kreipiant demesio j lytj; taip pat reikia palaips- 
niui mazinti saliskumLj moterij atzvilgiu, pa- 
sirasyti jmonese kolektyvines sutartis, rengti 
susitikimus ir diskusijas siuo klausimu. 

Perzvelgus literature apie tokio tipo socia- 
linip projektij ivertinirmj, atsizvelgiant j lyciij 
lygybes aspekttj, pastebeta, kad tokia politine 
veikla kartais vykdoma Ispanijoje (Astelarra, 
2005). Pasitaike atvejij, kai buvo vertinami 


• Public institutions with expertise in 
equality policies should promote more sys¬ 
tematic studies on womens situations, having 
some evaluation projects start from the wom¬ 
en’s needs, recognising them as a social group 
with their own specificities and with their 
own vulnerabilities in relation to situations of 
unemployment. 

• Indicators regarding gender should be 
taken into account to detect reduction of dis¬ 
parities between men and women. 

• In addition to experts in equality poli¬ 
cies involved in the project, every person who 
might influence a program should receive 
training both about advantages of integration 
of equality in those programs and about tools 
to realise. 

• The high heterogeneity among women 
requires priority to the development of pro¬ 
grams impeding the creation of marginal 
groups of women with low employability. 
Personal itineraries should be acknow¬ 
ledged as a keystone for success in women 
employability. 

• Female business entrepreneurship 
should be promoted with funds for creation 
of companies, management improvement, 
inspiration of confidence to beginning busi¬ 
nesswomen promoters, and involvement of 
technicians supporting enterprises owned by 
women. 

• Development of support for labour 
and familial conciliation, a larger awareness 
of businesspeople to recruit with no gen¬ 
der distinction and under the same condi¬ 
tions, and a gradual reduction of bias dis¬ 
favouring women by means of collaborative 
agreements at the enterprise level, as well 
as meetings and discussion forums in this 
regard. 

We have also reviewed the literature about 
evaluation of this type of social project from 
a gender perspective. We have noted infre¬ 
quent implementation of such policies in 
Spain (Astelarra, 2005). There have been some 
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su lygiomis galimybemis susijy projektai, ku- 
riuos vykde regionines valdzios institucijos, 
bei keletas aktyviij darbo politikos programij 
(Astelarra, 2001). 

2 

MOKYMO IR UZIMTUMO 
PROGRAMA: KONTEKSTAS,TIKSLAI 
IR METODOLOGINETYRIMO 
STRUKTURA 

2.1. KONTEKSTAS 

Autorii} atliktu tyrimu buvo jvertintos visos 
uzimtumo ir mokymo programos, kurias siu- 
lo Uzimtumo mokykla (UM), skirtos moterij 
uzimtumui gerinti. Viena jq aprasyta siame 
straipsnyje. Moterims buvo patariama, kaip 
pagerinti dalyvavimq darbo rinkoje naudo- 
jant informacines technologijas (IT). Progra¬ 
mos dalyvems suteikta informacija apie dar- 
bcj, teikiamos karjeros planavimo paslaugos, 
patariama, kaip ieskoti darbo, supazindinama 
su savarankisko darbo strategijomis ir skati- 
nama naudotis IT. Programa buvo siekiama 
gerinti moterij uzimtumq tiek kaimo, tiek ir 
miesto vietovese, taip pat labiau jtraukti jas 
j socialinj gyvenimq. 

2.2.1VERTINIMOTIKSLAI 

Savo vertinamuoju tyrimu siekeme „nustaty- 
ti, kokiomis sqlygomis ir aplinkybemis butq 
galima pagerinti moterij isidarbinamumcj, ir 
kaip tas s^lygas padaryti lengviau supranta- 
mas“. Tikslui pasiekti reikejo: 

1. Sukurti duomenij rinkimo sistem^. 
Tai daugiausia buvo anketos, kaip progra¬ 
mos dalyves ir personalas supranta paramij 
moterims. 

2. Imtis tam tikrq veiksnuj (pvz., issamus 
interviu su personalu, programos dalyvemis 
ir administracija, programos dalyvii] disku- 
sijos), leidzianciij pamatyti spragas moterij 


previous experiences evaluating equality proj¬ 
ects implemented by regional governments 
and some active labour policies programs 
(Astelarra, 2001). 

2 

A PROGRAM FORTRAINING 
AND EMPLOYMENT: 
CONTEXTUALISATION, PURPOSES 
AND METHODOLOGICAL DESIGN OF 
THIS STUDY 

2.1. CONTEXT 

The study carried out by the authors evaluated 
all the employment and training programs of 
SE, a public institution. It had developed them 
to promote the employment of women. One 
of these programs is reported here. It provided 
personal advice to facilitate labour participation 
using Information Technologies (IT). The pro¬ 
gram offered labour information, professional 
orientation, techniques to search for a job, strat¬ 
egies for self-employment and encouragement 
in the usage of IT. Through these actions, the 
program aimed to improve women employment 
both in rural and urban settings. A larger pres¬ 
ence of women in social life was also pursued. 

2.2. AIMS OF THE EVALUATION 

Our evaluation research at SE aimed “to iden¬ 
tify and facilitate understanding of conditions 
and circumstances which allow and constrain 
employability of women.” We operationalised 
this objective as follows: 

1. To design a system to collect informa¬ 
tion - mainly questionnaires - particularly 
users and staff perceptions of the support for 
women. 

2. To implement procedures - such as 
in-depth interviews to staff personnel, us¬ 
ers and administrators as well as discussion 
groups with users - to identify “gaps” in the 
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jsidarbinimo gerinimo politikoje ir nustatyti 
jij priezastis. 

3. |vertinti program^ is lyciq lygiq gali- 
mybiq perspektyvos, vadovaujantis aptartais 
principals. 

Todel mes, t. y. universitetine tyriimj gru- 
pe, sukureme isorinio jvertinimo sistemq Uz- 
imtumo mokyklos siulomoms programoms. 
Naudojome kiekybinius ir kokybinius tyrimo 
metodus (Chaudhuri, Vos, 1988; Lelitonen, 
Pahkinen, 1996; Levy, Lemeshow, 1999). Pa- 
rodeme, kad gerai suprantame programos 
apimamq teritorijq ir skirtings kontekstq. 
Programa vykdoma jvairiose miesto ir kaimo 
vietovese, esanciose arciau ar toliau nuo di¬ 
delip miestp ir 1.1. Del skirtingo konteksto ir 
ribotos straipsnio apimties cia apraseme tik 
anketinius duomenis. 

2.3. DARBO ETAPAI 

fvertinimq pradejome nuo duomenij rinkimo 
ir analizavimo. Per pirmqjq programos apz- 
valgq surinkti jvairus reikalingi dokumentai: 
jstatymai, veikos ataskaitos, strukturiniai 
duomenys, procesij, susijusiq su moterij ste- 
besena, rezultatai, sav^s jvertinimo ataskaitos 
ir informacija apie formalias iniciatyvas. 

Kiekybiniam jvertinimui pasirinkome 
statistiny dviejij anketiniq apklausp anali- 
zq. Viena anketa buvo skirta su programa 
dirbanciam personalui, kita - programos 
dalyvems. Atsizvelgiant j pagrindines musp 
tyrimo strukturines savybes, analizei pa- 
sirinkti sie metodai: aprasomoji statistika, 
Somerso D koeficientas, Gamma testas ir 
Coheno Kappa koeficientas. Taip galejome 
susidaryti bendrp vaizdp apie zmones, su- 
sijusius su programa, surinkome duomenp, 
kaip programa veikia, kaip valdoma ir kokia 
jos numanoma nauda. Pagrindiniai tyrimo 
etapai: 

1. Anketp detalizavimas. Pirmiau- 
sia isskyreme visoms uzimtumo ir moky- 
mo programoms tinkancias kategorijas. 
Tada pasirinkome su aprasoma programa 


improvement policies for employability of 
women and the reasons for those gaps. 

3. To evaluate the program from a gender 
perspective according to the principles dis¬ 
cussed above. 

Thus, we, a university-based research 
group, developed an external evaluation of the 
programs developed by SE. Finally, we used 
both quantitative and qualitative methods 
(Chaudhuri, Vos, 1988; Lelitonen, Pahkinen, 
1996; Levy, Lemeshow, 1999). We demonstrat¬ 
ed an extensive comprehension of the territory 
the program covers, coping with different pro¬ 
grammatic contexts. The program is imple¬ 
mented in a wide variety of locations - rural 
and urban; close to big cities and distant, etc. - 
bringing out an impact on all of them. Due to 
that diversity of contexts and the because of 
the length of this paper, we describe here only 
the questionnaire data. 

2.3. WORK PHASES 

We started this evaluation process with collec¬ 
tion and analysis of data. Our first review of this 
program provided documental information 
about this program in every area: legislation, 
activity reports, organisation, outcomes of the 
monitoring processes of women, self-evalua¬ 
tions, and information of formative initiatives. 

For the quantitative evaluation we made 
a statistical analysis of two questionnaires, 
one for the program staff, another for the us¬ 
ers. Such analysis consisted of descriptive sta¬ 
tistics, also Somers D, Gamma test & Cohens 
Kappa as the most appropriate inferential tests 
according to the major features of our design. 
This yielded a general prospective of the people 
involved as well as information about how the 
program worked, how it was managed and its 
perceived efficiency. Here are the major stages 
carried out: 

1. Elaboration of the questionnaires. 

We started from a set of categories for all em¬ 
ployment and the training programs. Then we 
chose issues pertinent to the program discussed 
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susijusius aspektus ir pagal juos parengeme 
bendrq anketos strukturq (1 lentele). Mes 
apdorojome is atskirq programos dalyviq 
gautq informacijq, naudodami tuos pacius 
saltinius ir priemones, ir kartu surinko- 
me konkrecius duomenis apie kiekvienq 
program^. 

Kiekviena anketa buvo padalyta j tris da¬ 
bs: (1) jvadq, kuriame pateikta informacija 
apie tikslus ir paaiskinta, kaip uzpildyti an- 
ketq; (2) demografinius respondent^ duome¬ 
nis: vietov^, centrq, issilavinimq ir profesijq; 
ir (3) aspektus, atskleidziancius respondent^ 
poziurj j program^. 


here and adapted the general structure of the 
questionnaire (Table 1). We triangulated in¬ 
formation from different users when the same 
resources and facilities were employed and, at 
the same time, collected specific information 
of every program. 

Each of the two questionnaires was divid¬ 
ed into three parts: (1) introduction, with in¬ 
formation about our purposes and how to fill 
out the questionnaire; (2) demographic data to 
identify respondents globally - location, cen¬ 
tre, academic and professional background; 
and (3) items indicating respondent perspec¬ 
tives of the program. 


1 lentele. Kiekvienai apklausoje dalyvavusiai grupei priskirtos kategorijos ir aspektai 
Table 1. Categories and issues assigned to every group surveyed 


KATEGORIJOS 

ASPEKTAI 

PERSONALAS 

DALYVES 

CATEGORIES 

ISSUES 

STAFF 

USERS 


Programos tikslq jgyvendinimas 

Viability of the program's objectives 

X 



Lemiami veiksniai 

X 



Determinants 



Logiskas turinio isdestymas 

Logical arrangement of content 

X 

X 


Medziagos charakteristika 

Characteristics of materials 

X 

X 


Programos tinkamumas 

Applicability of the program 

X 



Programos efektyvumas 

Program effectiveness 

X 

X 

A. Programa ir procesas 
A. Program and process 

Programos personalo taikytos metodologijos 
efektyvumas 

X 

X 


Effectiveness of the methodology employed by the staff 
personnel of the program 


Grjztamoji programa 

Return program 

X 

X 


Nenumatytas pasalinis poveikis 

Unanticipated side effects 

X 



Proceso nuoseklumas 

Consistency of the process 

X 

X 


Proceso atitikimas tikslams 

Adequacy of process to objectives 

X 



Norimq tikslq pasiekimas 

Attainment of desired goals 

X 

X 
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KATEGORIJOS 

CATEGORIES 

ASPEKTAI 

ISSUES 

PERSONALAS 

STAFF 

DALYVES 

USERS 

B. Tikslines personalo 
ir programos 
dalyvii} grapes 

B. Targeted users 
and staff 

Programos pritaikymas pagal poreikius 

Adaptation of the program to the needs and demands 

X 

X 

Programos turinio panaudojimas 

Operation of program content 

X 


Tinkama veikla 

Suitability of activities 

X 

X 

Siulomos metodologijos efektyvumas 

Effectiveness of the methodology proposed 

X 

X 

Programos poveikis dalyvems nuo pradzios iki pa- 
baigos: energingumas, motyvacija, lukesciai, proceso 
jvertinimas, programos rezultatai ir sekme pagal 
turetus lukescius 

Effects of the program on users from beginning to end: 
driving attitude, motivation, expectations, evaluati¬ 
on of the process, outcomes, success of the program 
depending on the level of expectations 

X 

X 

Program^ taikancio asmens vaidmenys 

Roles of the person applying the program 

X 

X 

Pasitenkinimas 

Satisfaction 

X 

X 

Tvarkarastis ir programos pradzios laikas 

Timetable and time of application of the program 

X 

X 

Problemos ir sunkumai, susij^ su organizaciniais, 
veiklos ir funkcionalumo aspektais 

Difficulties, problems in the organizational, operatio¬ 
nal and functional levels 

X 


Mokymo poreikiai 

Training needs 

X 

X 

C. Jslaiga 

C. Agency 

Tinkamumas kontekstui 

Suitability for the application context 

X 


Programos poveikis atitinkamoje srityje 

Program’s impact on the area 

X 


Kaip program^ vertina pati jstaiga 

Valuation of the program by the agency 

X 


Rysys su kitomis panasiomis paslaugomis 

Relationship with other similar services 

X 



Galutinius anketij variantus patvirtino 
ekspertai - asmenys, ismanantys su progra- 
ma susijusias sritis, taciau nepriklausantys 
jvertinimq atliekanciai komandai. Patvirtini- 
mas reiske, kad pasirinkta nuosekli priemo- 
ne, t. y. svarbus aspektai ir tinkami tiriamos 
temos atstovai. Pasinaudojome Amerikos 


Once a final draft of every questionnaire 
was ready, experts validated them all. We turned 
to people familiar with every field each program 
coped with who were external to the evaluative 
team. The experts’ validation provided consis¬ 
tency to the tool employed, mainly about the 
relevance of items and the representatives of the 
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psichologq asociacijos (APA) pateiktomis 
techninemis rekomendacijomis (Feigl, Scri- 
ven, 1976) del vadinamijjq jprasto pagrjs- 
tumo teorijq. Tiksliau kalbant, ememes sip 
veiksmij: 

• Apibrezeme stebejimij visumq. 

• Nustateme nagrinejamij objektq srities 
ekspertus. 

• Vadovaudamiesi atitinkama struktura, 
pateikeme ekspertij nuomon^ apie anketos 
turinio svarbq ir tai, kaip ji atspindi nagrine¬ 
jamij objektq visumq. 

• Pristateme procedure, kaip apiben- 
drinti aprasytij veiksmij rezultatus. 

Taip pat nustateme anketij patikimumcj, 
kad gautume panasius rezultatus, isdalijf jas 
lygiavertemis sqlygomis. Patikimumui anali- 
zuoti surinkome homogeniskumo ir vidinio 
suderinamumo indeksus. Aspektai, turintys 
mazq santyki su visuma, buvo atmesti. Ap- 
skaiciavome Cronbacho Alpha koeficientq 
kiekvienai kategorijai, apgalvoj^, ar tam tik- 
rus trukdancius atsakymus palikti, ar atmesti. 
Darant prielaidq, kad 0,8 ir didesnis santy- 
kis yra labai geras, gautas 0,892 Cronbacho 
Alpha koeficientas rodo labai aukstcj vidinj 
suderinamumq. 

2. Tyrimo imties parinkimas. Pasi- 
rinkome Mastering atrankij, kad pogrupis 
atspindetij kiekvienij programoje dalyvau- 
janciij grupq. Tyrimo imtis buvo pritaikyta 
pagal tikslines tiriamijjij grupes, programos 
dalyvius ir personal^. Gauta galutine imtis 
nurodyta 2 ir 3 lentelese. 

3. Medziagos rinkimas. Anketij naudo- 
jimas. Anketos isdalytos kiekvienoje vietoje, 
kur vykdyta programa. Dalyves, kurias del 
geografiniij priezasciij buvo sunku pasiekti, 
apklauseme telefonu, naudodami programing 
jrangp (Computer-Assisted Telephone Intervie¬ 
wing, CATI). Taip galejome rinkti duomenis 
ir tuo pat metu juos automatiskai koduoti bei 
analizuoti. 

4. Statistine analize. Statistinei ana- 
lizei pasinaudojome Statistines analizes ir 


topic enquired. As for the so called conventional 
theory of validity, we kept in mind the Ameri¬ 
can Psychological Association (APA) techni¬ 
cal recommendations (Feigl & Scriven, 1976). 
More in particular, we proceed as follows: 

• Definition of the universe of acceptable 
observations. 

• Identification of experts of that 
universe. 

• Experts’ judgements about to what ex¬ 
tent the content of the questionnaire is relevant 
and representative of that universe by using a 
structured procedure. 

• A procedure to summarise the out¬ 
comes of the prior step. 

We also determined reliability of the ques¬ 
tionnaires to find out their capacity to provide 
similar results when passed out in equivalent 
conditions. To analyse reliability we got the in¬ 
dices of homogeneity and internal consistency. 
Items with a low relation with the total were 
discarded. We also performed the analysis of 
scales by using the Cronbach’s Alpha test for 
every category, reconsidering pertinence or 
exclusion for disruptive items. Given that rates 
from 0.8 are considered very good, the Cron- 
bach’s Alpha index we obtained, 0.892, denotes 
a very high internal consistence. 

2. Sample selection. A cluster sampling 
was carried out so that we worked with a sub¬ 
set of the population which was representa¬ 
tive for each group involved in this program. 
Our sample was customised for every targeted 
population: users and staff personnel. The final 
sample was taken out of Tables 2 and 3. 

3. Fieldwork. Application of question¬ 
naires. The application of questionnaires was 
face to face in every site where the program 
has been implemented in. For subjects consid¬ 
ered as a priority for this evaluation who were 
difficult to access for geographical reasons, we 
employed a Computer-Assisted Telephone In¬ 
terviewing (CATI) software. This procedure 
allowed to collect data and, at the same time, 
to codify and analyse them automatically. 
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2 lentele. Programos dalyvii) imties duomenys 


Table 2. Users sample data producer 


PROGRAMOS DALYVES / USERS 

KLASTERIS 

CLUSTERS 

VIETOViy SKAICIUS 

NUMBER OF LOCATIONS 

DALYViy SKAICIUS VIETOVEI 

SUBJECTS PER LOCATION 

DALYViy SKAICIUS SRICIAI 

SUBJECTS PER AREA 

IS VISO 

TOTAL 

1 

2 

17 

20 

37 

356 

2 

3 

23 

12 

19 

54 

3 

3 

11 

14 

16 

41 

4 

3 

19 

16 

22 

57 

5 

2 

13 

16 

29 

6 

3 

19 

20 

14 

53 

7 

2 

3 

10 

13 

8 

4 

17 

17 

21 

17 

72 


duomenq apdorojimo (SPSS) programme 
jranga (14.0 versija). Ja gauti sie duome¬ 
nys: (1) dazniq ir procentinis pasiskirsty- 
mas; (2) aritmetinis vidurkis ir mediana; 
(3) nuokrypis; (4) juostines ir skritulines 
diagramos. 

Somerso D koeficientu buvo isskirti ne- 
priklausomi ir priklausomi kintamieji. Siuo 
testu nustatoma priklausomybe tarp dviejq 
kintamqjij (intervale nuo -1 iki 1). Kuo ko- 
efkientas arciau 1 (t. y. absoliucios vertes), 
tuo stripresne dviejq kintamqjij priklau¬ 
somybe. Ir atvirksciai - kuo verte arciau 0, 
tuo priklausomybe silpnesne. Sj testcj. pasi- 
rinkome kaip tinkamq priemon^ koreliacijai 
tarp dviejq kintamijji} atsitiktinese lentelese 
aprasyti. 

Priklausomiems kintamiesiems taip pat 
naudojome Gamma koeficient^ (intervale 
nuo -1 iki 1). Kuo koeficientas arciau 1, tuo 


4. Statistical analysis. We employed the 
Statistical Package for the Social Sciences 
(SPSS), version 14.0, for the statistical analysis. 
First, we use this software to get some descrip¬ 
tive statistical tests, such as: (1) distribution of 
frequencies and percentages; (2) arithmetic 
mean and median; (3) deviation; and (4) bar 
and pie charts. 

We used the Somers’ D to cross indepen¬ 
dent and dependent variables. This test mea¬ 
sures the association between two variables 
which takes a value between -1 and 1. The 
closer the test is to 1 - bsolute value, - the stron¬ 
ger is the relationship between variables. The 
closer to 0, the lower the relation is. We em¬ 
ployed this test as a suitable tool to describe 
the association between the two variables of 
our contingency tables. 

We used the Gamma test to cross de¬ 
pendent variables, ranging from -1 to 1 too. 
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3 lentele. Personalo imties duomenys 
Table 3. Staff personnel sample data producer 


PERSONALAS . STAFF PERSONNEL 

KLASTERIAI 

CLUSTERS 

VIETOViy SKAICIUS 

NUMBER OF LOCATIONS 

DALYViy SKAICIUS VIETOVEI 

SUBJECTS PER LOCATION 

DALYViy SKAICIUS SRICLAI 

SUBJECTS PER AREA 

IS VISO 

TOTAL 

1 

2 

1 

2 

3 

33 

2 

3 

2 

3 

3 

8 

3 

3 

2 

2 

4 

4 

3 

1 

1 

2 

4 

5 

2 

1 

1 

2 

6 

3 

1 

1 

1 

3 

7 

2 

1 

2 

3 

8 

4 

1 

2 

1 

2 

6 


didesne priklausomybe. Jei gauta verte ar- 
timesne 0, priklausomybes tarp kintamijji} 
visai nera arba ji labai nedidele. Sj test^ pasi- 
rinkome noredami ismatuoti priklausomyb^ 
tarp suskirstyti} duomeni}, kadangi abu kin- 
tamieji matuojami rangu grupeje. O tai ir yra 
pagrindinis sio testo tikslas. 

Abu testai pasinaudoja range} informaeija 
linijiniam rysiui nustatyti. {prastesni normi- 
niai testai tokius galimus rysius daznai pra- 
leidzia. Teigiama verte reiskia tiesioginj rysj 
tarp kintamijjij, t. y. kuo didesnis vienas kin- 
tamasis, tuo didesnis ir kitas. Ir atvirksciai - 
neigiama verte reiskia atvirkstinj rys|, t. y. 
kuo didesnis vienas kintamasis, tuo mazesnis 
bus kitas. Pasinaudojome ir 0,05 kritine verte, 
kad galetume atmesti nepriklausomybes hi- 
potez^ ir daryti prielaid/}, kad visi analizuoti 
kintamieji yra susij^. 


The closer the test value is to 1, the stronger 
the relationship is. A value close to 0 implies 
a scarce or null relation between variables. We 
opted for this test to measure the strength of 
association of the cross tabulated data, since 
both variables are measured at the ordinal lev¬ 
el, and this is the major purpose of this test. 

These two tests take advantage of this 
ordinal information to allow the establish¬ 
ment of lineal relations, whereas the more 
usual nominal tests overlook such potential 
relationships. That is, a positive value implies 
a direct relationship between variables - the 
higher one is, the higher the other too. Con¬ 
versely, a negative value supposes an inverse 
relation - the higher one is, the lower the other. 
Also, we took a critical value of 0.05 to reject 
the independence hypothesis and assume that 
variables analysed were related. 
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Galiausiai suskaiciavome Coheno Kappa 
koeficientq, t. y. sia statistine suderinamumo 
priemone nustateme, kaip sutampa persona- 
lo ir programos dalyviij rezultatai. Is 1 lente- 
les matyti, kad lygindami grupii} rezultatus 
abejose anketose panaudojome bendrus pri- 
klausomus kintamuosius, tik kalbin^ raiskg. 
pritaikeme kiekvienai grupei. 

3 

PROGRAMOS JVERTINIMO REZULTATAI 

3.1. DUOMENYS APIE RESPONDENTUS 

Daugiausia vertinamos programos daly- 
viq priklause 31-40 metq (40 %) amziaus 
grupei. 28 % respondenciq 41-50 metq, o 
23 % - 30 metq ir jaunesnes. Tik 5 % ty- 
rime dalyvavusiij moterij neturi islaikyti- 
niq. Vidutiniskai vienai programos dalyvei 
priklauso 2,1 islaikytiniq. Tai gana aukstas 
rodiklis, palyginti su kitomis tos pacios js- 
taigos uzimtumo ir mokymo programomis. 
Susikertantys veiksniai rodo tokkj priklau- 
somyb^ tarp amziaus ir islaikytiniij skai- 
ciaus: kuo vyresne programos dalyve, tuo 
daugiau ji turi islaikytiniij. 

53 % sios imties moterij - bedarbes. Sis 
rodiklis butq 15 % didesnis, jei siai grupei 
priskirtume ir namij seimininkes, t. y. mo- 
teris, kurioms uz darbq nemokama. Apmo- 
kamas darbas (jskaitant darbdavius ir versli- 
ninkus) sudaro 18 %. Lik^ 14 % respondentij 
pasirinko atsakymcj „kita“. Mazdaug trecdalis 
respondenciij turi mokyklos baigimo atesta- 
tcj, dar beveik trecdalis - bakalauro ar profe- 
sinio rengimo diplomLj, 14 % - mokyklos bai¬ 
gimo pazyme jimLj. 7 .18 % moterij turi aukst;yi 
issimokslinimcj. Dauguma respondenciij yra 
menko issimokslinimo. 


7 Ispanijoje moksleiviai, negav^ mokyklos baigimo atestato, 
gauna pazymejim^, jrodantj, kad yra baig^ privalomojo ug- 
dymo program^. 


Finally, we also employed the Cohens Kappa 
coefficient, that is, we used a statistical measure 
of concordance for categorical items to deter¬ 
mine to what extent the staff personnel and the 
users agreed. We used common dependent vari¬ 
ables in both questionnaires to allow this com¬ 
parison, as described in table 1, but adapting the 
language of those shared items for each group. 

3 

THE RESULTS OF THE PROGRAM 
EVALUATION 


3.1. RESPONDENTS CHARACTERISTICS 

The age range encompassing the largest num¬ 
ber of users of the program evaluated is that 
of 31-40 years (40%), followed by 41-50 (28%) 
and then 30 or less (23%). Only 5 % of wom¬ 
ens in the sample have no dependents, being 
the average 2.1 per user. This rate is high com¬ 
pared with other employment and training 
programs of the same agency. Crossing factors 
determine that association between age and 
number of dependents, as it could be expected 
in anticipation: the older the user is, the more 
dependents on her. 

Women unemployed in this sample are 
53%. This rate should be increased a 15% more 
if we take into account those women who are 
housewives, that is, an unpaid work. Percent¬ 
age of paid work, comprising both employers 
and entrepreneurs, is 18%. The rest, almost a 
14% declared other’ situation. Around one 
third has a School Graduation, nearly another 
third a Baccalaureate or a Professional Train¬ 
ing degree, and 14% a School Certificate 7 . The 
percentage of women with a higher education 
degree is 18%. By and large, the academic level 
of the sample is low. 


7 In Spain, students who do not get the school degree re¬ 
ceive this school certificate of completion of compulsory 
schooling. 
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3.2. PAGRINDINIAI VARTOTOJg IR 
DARBUOTOJg ATSAKYMy REZULTATAI 

Is apklausos rezultatq matyti, kad vartotojai 
labai domisi programa. 1,8 % respondent^ 
pateike neigiamq atsakymq („mazai domi¬ 
si" ar „visai nesidomi"), o 98,2 % - teigiamq 
(„pakankamai domisi" - 36 %, „labai do¬ 
misi" - 62,2 %). 

Taip teigiamai atsakyta beveik j visus tei- 
ginius. Palygin^ sios jstaigos siulomas pro- 
gramas, matome, kad cia aptarta programa 
vertinama geriausiai. Toliau pateikti keli ge- 
riausiai jvertinti teiginiai. 

Vienintelis teiginys, sulaukqs neigiamo 
jvertinimo, - „si paslauga man padejo susi- 
rasti darbq". Respondentai galejo pasirinkti 
atsakymus „taip“ arba „ne“ ir 68 % apklaus- 
tqjq, susiradusii} darbq, j sj klausimq atsake 
„ne“. Mazdaug du trecdaliai tip, kurioms si 
paslauga padejo rasti darbq, buvo jdarbintos, 
o likusios - laisvai samdomos. 

Pagal kintamqjq susikirtima matyti, jog 
zemiausias nedarbo lygis yra tarp moterij, 


3.2. MAJOR FINDINGS WHEN ANALYSING 
USERS'AND STAFF'S REPLIES 

Users’ interest for the program is very high, 
according to the answers given in the ques¬ 
tionnaire to this item. 1.8% provides a nega¬ 
tive answer (“little” or “null”), 98.2% positive 
(“enough”, 36%, “a lot”, 62.2%). 

This positive answer is predominant in al¬ 
most all the items. In fact, if we look at all the 
programs this agency offers, we find that this one 
discussed here is one of the best valued. These 
are some of the items with the highest scores. 

The only item with a negative answer is 
I have found a job through this service, with yes’ 
and no’ as possible answers, which arose a 68% of 
negative answers among those users who found 
a job. From those who found a job through this 
service, around two-thirds were working as em¬ 
ployed person, the rest as freelance. 

If we pay attention to the intersection bet¬ 
ween variables, we observe that the group of 
women with a higher degree qualification has 
the lowest rate of unemployment. Likewise, 


4 lentele. Dalyviq geriausiai vertinami teiginiai 


Table 4. Items best valued by users 


TEIGINIAI 

ITEM 

PRITARIU / LABAI PRITARIU (%) 

AGREE / STRONGLY AGREE (%) 

Is esmes esu patenkinta personalu 

By and large, I am pleased with the personnel staff 

97,4 

Paslauga teikiantis personalas yra susipazin^s su kliutimis ir diskriminaci- 
ja, kuri^ moterys patiria darbe 

The staff attending the service knows the obstacles and discrimination that 
women suffer at work 

96,8 

Pateikta aiski ir gerai organizuota informacija 

Information received is clear and well-organised 

93,9 

Paslaugos teikimo valandos man atrodo tinkamos 

The operating hours of this service seem right to me 

92,2 

Vartojama neseksistine kalba 

A non-sexist language is employed at any time 

91,4 

Galiu vykdyti duotus nurodymus ir patarimus 

I am able to implement the instructions and guidance received 

91,2 

Mane tenkina, kaip vystoma teikiama paslauga 

I am satisfied with how it is developing the service 

90,6 
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turinciij geresnj issimokslinimy Ir atvirks- 
ciai - prastesnis issimokslinimas rodo di- 
desnj nedarbo lygj. Gauti duomenys taip pat 
atskleide, kad moterij, turinciij aukstyjj issi- 
mokslinimy ir turinciij bakalauro ar profe- 
sinio mokymo diplomy uzimtumo lygis sky- 
resi maziau sudalyvavus sioje programoje. 

Pastebetina, kad 41-50 metij moterys pa- 
tiria daugiau sunkumij ieskodamos darbo per 
sicj program^. Zemiausias nedarbo lygis jau- 
nesniij nei 30 metij amziaus grupeje. Todel 
aiski tendencija - kuo vyresne moteris, tuo 
sunkiau jai rasti darby 

3.3. RYSYSTARP AMZIAUS, PROFESIJOS, 
MOKYMO IR SUSIDOMEJIMO 
PROGRAMA 

Priklausomybei nustatyti panaudoti Somer¬ 
set D ir Gamma testai parode: vyresnes tyri- 
mo dalyves geriau jvertino teiginj „persona- 
las suteikia priemoniij, leidzianciij tobuleti". 
Be to, turima profesija pozityviai koreliuoja 
su dauguma teiginiij. Galima daryti prielaidy 
kad bedarbes moterys ir nanuj seimininkes 
si;j paslaug;j vertino prasciau. Kitaip tariant, 
susiradus darb;j programa jvertinta geriau. 

Panasiai daugelis teiginiij susieti su kin- 
tamuoju „susidomejimas programa". Rasta 
net 16 is 24 galimij asociaciji}. Tai uzuomina, 
jog programos dalyviij lukesciai didzia dalimi 
pateisinami baigus program^. „Mokymosi re- 
zultatai" taip pat neigiamai siejami su dauge- 
liu is aspektij. Kuo geresnis issimokslinimas, 
tuo mazesnis rezultatas skiriamas tirtiems 
teiginiams, isskyrus viencj (t. y. programa 
„suteikia naujij minciij ir idejij"), turintj tei- 
giamas SLjsajas. 

3.4. PROGRAMOS DALYVig IR 
PERSONALO ATSAKYMg SUTAPIMAS 

Remiantis Coheno Kappa testu, programos 
dalyviij ir personalo atsakymai j daugumcj 
pateiktij teiginiij sutampa. Gautij rezultatij 
santrauka pateikta 5 lenteleje. 


the less educational level, the higher the un¬ 
employment rate. Data also show that differ¬ 
ences of employment rates between women 
with a higher education degree and Baccalau¬ 
reate or Professional training degrees are lower 
once the users take part in this program. 

As for the age of participants, the 41-to-50- 
year-old group experiences more difficulties to 
find employment through this program. At the 
same time, the less-than-thirty group has the 
lowest rate of unemployment. So there is a clear 
tendency: the older, the less likely to find a job. 

3.3. RELATIONS FOUND AMONG AGE, 
OCCUPATION,TRAINING AND INTEREST 
FORTHEPROGRAM. 

Measures of association - Somers’ D and gam¬ 
ma tests - also determined that the older the 
users are, the higher appraisal of the item the 
personnel staff provides tool for improvement. 
Also, current occupation relates to most of the 
items positively. Our interpretation for this as¬ 
sociation is that unemployed and housewives 
valued the service worse or, in other words, if 
you find a job you value the program higher. 

Likewise, many of the items are associat¬ 
ed with the variable interest in the program, as 
can be found in 16 out of 24 existing associa¬ 
tions. This hints that users’ expectations are 
fulfilled at large after the completion of the 
program. Educational attainment is also asso¬ 
ciated negatively with most of the items; the 
higher the degree, the lower the score given 
to those items, except for the item new ideas 
are provided in the program, with a positive 
association. 

3.4. LEVEL OF AGREEMENT BETWEEN 
USERS'AND PERSONNEL STAFF'S 
ANSWERS. 

According to the Cohen’s Kappa test, there is 
a high agreement between users’ and staff’s an¬ 
swers for the vast majority of common items. 
Below is a summary of such results (Table 5). 
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5 lentele. Programos dalyviq ir personalo atsakymq sutapimas 


Table 5. Level of agreement between users and personnel staff 


KINTAMIEJI 

KAPPA 

SUTAPIMO LYGIS 

VARIABLES 

LEVEL OF CONCORDANCE 

Naudojami kompiuteriai 

0,962 

Labai geras 

Computers are used 

Very good 

Naudojama neutrali kalbine raiska 

0,662 

Geras 

An inclusive language is employed 

Good 

Pasitenkinimas programos pletra 

0,661 

Geras 

Satisfaction with program development 

Good 

Programa duoda rezultatus, kuriq tikimasi 

0,809 

Labai geras 

The program gives the expected results 

Very good 

Programa pateisino dalyviq lukescius 

0,867 

Labai geras 

The program has responded to the users’ expectations 

Very good 

Veiksmai atitiko tikslus 

0,776 

Geras 

Actions have been consistent with objectives 

Good 

Programa atitinka dalyviq poreikius 

0,583 

Vidutiniskas 

The program has responded to the users’ needs 

Moderate 

Tinkamas programos turinys 

0,526 

Vidutiniskas 

Contents are appropriate 

Moderate 

Tinkamos uzduotys 

0,551 

Vidutiniskas 

Activities are appropriate 

Moderate 

Programa praplecia dalyviq karjeros galimybes 

0,976 

Labai geras 

The program broadens the career choices of users 

Very good 

Programos dalyviq ir personalo pasitenkinimas programa 

0,886 

Labai geras 

Satisfaction between users and staff of the program 

Very good 

Paslaugos teikimo valandos tinka 

0,866 

Labai geras 

Operating hours of the service is appropriate 

Very good 

Techninio personalo zinios apie sunkumus, su kuriais moterys susi- 
duria darbo vietoje 

0,946 

Labai geras 

Technical staff knowledge of women difficulties in the workplace 

Very good 

Vidurkis 

0,770 

Geras 

Average 

Good 


Atlikta analize parode, kad dvi grupes, 
jvertintos „gerai“, sutampa pagal gana aukstas 
Coheno Kappa testo vertes. 10 is 13 is bendrij 
teiginiij buvo jvertinti „gerai“ ar „labai gerai“. 
Ne vienas teiginys nepasizymejo zemu suta- 
pimo lygiu. Todel galima daryti isvadq, kad 
abieji} tirtij grupii} poziuriai j teiktq paslaugq 
sutampa. Si programa pasizymi geriausiu su- 
tarimu is visi} sios jstaigos siulomq uzimtumo 
ir mokymo prograimj. 


The analysis arises a level of agreement 
between the two groups tagged as good’ con¬ 
sidering the high values of the Cohens Kappa 
test. 10 out of 13 common items has good’ or 
Very good’ concordances with no low agree¬ 
ment in any item, so we can conclude that both 
collectives have a similar view of this service. 
This program actually reaches the highest level 
of agreement among all the employment and 
training programs this agency carries out. 




















Antonio BUSTOS, Jose Luis AROSTEGUI 
APMOKYMOIR UZIMTUMO PROGRAMOS (VERTINIMAS: STRUKTUROS APTARIMAS 
THE EVALUATION OF A TRAINING AND EMPLOYMENT PROGRAM: DISCUSSION ON DESIGN 185 


Svarbu priminti, kad testas nenurodo, 
kaip - gerai ar blogai - vertinamas tam tikras 
teiginys. Is jo isaiskeja, ar dviejq tirtij grupiij 
nuomones sutampa. 

ISVADOS 

Programij jvertinimas institucinio vertinimo 
kontekste atskleidzia potencialias galimybes, 
kurios prieinamos aukstojo mokslo tyrejams. 
{ tiriamyjj darby jtraukus technologijas ir 
procedures, galima padaryti teigiamy itaky 
priimant sprendimus, susijusius su viesyja so- 
cialine politika. Siame straipsnyje panaudota 
kiekybine metodologija leidzia pajvairinti 
analiz^ ir geriau suprasti tirtus dalykus bei 
priimti sprendimus, leidziancius pagerinti 
vertinamy programy. 

Tyrimas atskleide, jog reikia atlikti isori- 
nj programij vertinimy, leidziantj nustatyti, 
kaip buvo panaudotos viesosios lesos, o siuo 
konkreciu atveju - kaip jos buvo panaudotos 
lyciq lygioms galimybems skatinti. Kadangi 
pastaraisiais metais padaugejo investicijij j 
lygiij galimybiij politikos pletry, valdzios ir 
viesosios jstaigos turetq prisiimti etin§ atsa- 
komyb^ stebeti ir jvertinti veiksmus, kuriij 
buvo imtasi. 

Kalbant apie siame straipsnyje aptartos 
programos j vert ini my, galima pastebeti, kad 
jmanoma kokybiskai jgyvendinti uzimtumo 
ir mokymo programy, jeigu skiriamas deme- 
sys suinteresuotoms grupems. Pastebeta dide- 
le darna tarp su programa susijusios veiklos, 
programos dalyviij numanomij pasiekimij ir 
planuojamij veiksmij funkcionalumo. Jeigu 
programos dalyviai ir personalas sutaria del 
pagrindiniij dalykij, vadinasi, programa yra 
gerai organizuota, turi aiskiy strategijy, integ- 
ruotas lyciq demuo ir, galiausiai, pasitenkini- 
mas programos vystymu ir jos rezultatais yra 
didelis. 

Turint omenyje, kad vyresniems zmo- 
nems sunkiau rasti darby, galima daryti 


It is important to remind that this test does 
not determine a good or bad consideration of 
the item but, rather, the agreement in such a con¬ 
sideration between the two groups enquired. 

CONCLUSIONS 

The evaluation of programs in the context of 
institutional assessment is a demonstration of 
potentialities that higher-education research¬ 
ers can deploy. Incorporation oftechniques and 
procedures employed in research work can in¬ 
tervene positively in decision taking of public 
social policies. In the case reported in this pa¬ 
per, the quantitative methodologies presented 
in this paper offers versatility of analysis and 
interpretations that help understand the reali¬ 
ties explored and make decisions to improve 
the performance of the program evaluated. 

In particular, the study stresses the need to 
incorporate external evaluation of programs 
to give account of how public funds have been 
employed and, namely in this case, through 
a gender perspective. Given that public invest¬ 
ment in the development of equality policies 
have gained momentum in recent years, an 
ethical responsibility to conduct monitoring 
and evaluation of actions should be lived up to 
the budgetary efforts of government and pub¬ 
lic agencies. 

Speaking of the evaluation of the program 
discussed in this article, we find out how it is 
possible to implement with high quality an 
employment and training program conceived 
from the perspective of the groups involved. 
The degree of consistency in its operation, in 
the participants’ perception of achievements, 
and in the functionality of planned actions is 
very high. Level of agreement on basic issues of 
the program between users and personnel staff 
denote good organisation, clear strategies well- 
organised, a gender mainstreaming perspective 
and, by and large, a high satisfaction with the 
development and the outcomes of the program. 
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prielaidq: vyresni asmenys program^ verti- 
na blogiau nei jaunesni. Taciau jvertinimas 
atskleide, kad auksciausius jvertinimus sky- 
re butent sios amziaus grupes dalyves. Taigi 
gali buti, jog programa pritaikyta didziau- 
siems sunkumams, su kuriais siuo metu 
susiduria moterys, ieskodamos darbo. Vis 
delto amzius joms tebera lemiamas veiksnys 
ieskant darbo. Be to, dauguma programos 
dalyviij neturi universitetinio issimoksli- 
nimo. Todel akademine kvalifikacija yra 
dar viena svarbi sqlyga, galinti padeti rasti 
darbq. 

Galiausiai, tokio tipo vertinamasis ty- 
rimas padeda tobuleti ir aukstojo mokslo 
tyrejams, ir lektoriams. Institucinio jverti- 
nimo sudetingumas, issamumas ir papildo- 
ma veikla suteikia galimyb^ universitetams 
augti ir tobuleti mokslineje srityje. Vieso- 
sios jstaigos ir universitetai gali gauti abi- 
puses naudos, universitetai gali pasidalyti 
savo patirtimi ir net apmokyti tokiq pro- 
gramij teikejus diagnostiniais tikslais. Kartu 
tai puiki pakopa tyrimo metodij pletrai ir 
mokymuisi. 
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If we keep in mind that the older the person 
is, the more difficulties finding employment, it 
could be argued that older users perceive this 
program worst than their younger service fel¬ 
low. However, the evaluation process found that 
the highest ratings were given by this age group, 
so it could be that the program is adapted to the 
greatest difficulties that women have to face at 
present to find a job. The age is still a determin¬ 
ing factor to find a job for these women, though. 
In addition, a large part of users has no univer¬ 
sity degrees or undergraduate studies, being the 
academic qualification another clear condition 
for these women to find employment. 

Last, this type of evaluative research nur¬ 
tures instruction of researchers and lecturers 
in higher education. The complexity, depth 
and complementary treatments of institutional 
evaluations provide an opportunity for scien¬ 
tific growth in universities. In this exchange of 
benefits between public institutions and uni¬ 
versities, the latter can afford their experience 
and training for diagnostic purposes related to 
the evaluand. At the same time, this is an ex¬ 
cellent stage for development and learning of 
research methods. 
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